Image sensing apparatus and method of controlling the image sensing apparatus

ABSTRACT

A sensed image generated from an image signal output from an image sensor for receiving light that becomes incident sequentially through an imaging lens and a microlens array that is a two-dimensional array including a plurality of microlenses is acquired. A list is created in which, for each pixel position on the image sensor, the correspondence between the coordinates of the light incident at the pixel position on the imaging lens and the coordinates of the pixel position is registered. Images obtained by rearranging pixels at the coordinate positions on the image sensor corresponding to the coordinates in accordance with the arrangement order of the coordinates on the imaging lens are generated as a parallax image group.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image sensing technique.

2. Description of the Related Art

In a digital camera, conventionally, an object image is formed on animage sensing element through an imaging lens, and image informationobtained by the image sensing element is displayed on an LCD (LiquidCrystal Display). The digital camera includes an electronic viewfinder(EVF) to observe the image information formed on the LCD (JapanesePatent Laid-Open No. 5-134294).

Also known is a digital camera using live view display in which ashutter arranged on the whole surface of an image sensing element isopened to project an object image on the image sensing element, therebydisplaying the video in an image display LCD (Japanese Patent Laid-OpenNo. 2001-186401).

International Patent Publication No. 06/039486 and Ren. Ng, et al “LightField Photography with a Hand-Held Plenoptic Camera”, Stanford TechReport CTSR 2005-02 propose an image sensing apparatus using a methodcalled “Light Field Photography”. This image sensing apparatus includesan imaging lens, a microlens array, an image sensing element, and animage processing unit. Sensed image data obtained from the image sensingelement includes not only the light intensity distribution on thelight-receiving surface but also the information of the light travelingdirection. The image processing unit can reconstruct an image observedfrom a plurality of viewpoints or directions.

However, when the image sensing apparatus including the microlens arrayarranged in front of the image sensing element senses an object as shownin FIG. 20A using an aperture stop having a circular opening portion,the live view display image includes a non-light-receiving area, asshown in FIG. 20B. For this reason, it is difficult for the user todetermine the focus position.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of theabove-described problem, and provides a technique for converting a liveview display image into an image for which the user can easily determinethe focus position.

According to the first aspect of the present invention, an image sensingapparatus comprises: a unit that acquires a sensed image generated froman image signal output from an image sensor for receiving light thatbecomes incident sequentially through an imaging lens and a microlensarray that is a two-dimensional array including a plurality ofmicrolenses; a creation unit that creates a list in which, for eachpixel position on the image sensor, a correspondence between coordinatesof the light incident at the pixel position on the imaging lens andcoordinates of the pixel position is registered; a generation unit thatgenerates, as a parallax image group, images obtained by rearrangingpixels at pixel positions on the image sensor corresponding to thecoordinates in accordance with an arrangement order of the coordinateson the imaging lens registered in the list; and an output unit thatoutputs the images generated by the generation unit.

According to the second aspect of the present invention, a method ofcontrolling an image sensing apparatus including an image sensor forreceiving light that becomes incident sequentially through an imaginglens and a microlens array that is a two-dimensional array including aplurality of microlenses, comprises: a step of acquiring a sensed imagegenerated from an image signal output from the image sensor; a creationstep of creating a list in which, for each pixel position on the imagesensor, a correspondence between coordinates of the light incident atthe pixel position on the imaging lens and coordinates of the pixelposition is registered; a generation step of generating, as a parallaximage group, images obtained by rearranging pixels at pixel positions onthe image sensor corresponding to the coordinates in accordance with anarrangement order of the coordinates on the imaging lens registered inthe list; and an output step of outputting the images generated in thegeneration step.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of the arrangement of animage sensing apparatus;

FIG. 2 is a block diagram showing an example of the functionalarrangement of a signal processing unit 106;

FIG. 3 is a view showing an aperture stop 102;

FIG. 4 is a view showing the positional relationship;

FIG. 5 is a view showing an example of light-receiving areas for therespective microlenses on the light-receiving surface of an image sensor104;

FIG. 6 is a flowchart of processing to be performed by the signalprocessing unit 106;

FIG. 7 is a view showing examples of area division and the indices ofareas of an imaging lens 101 when m=3, and n=3;

FIG. 8 is a view for explaining coordinates (u,v) on the imaging lens;

FIG. 9 is a table showing an example of the arrangement of a lightfield;

FIG. 10 is a view for explaining a light field creation method;

FIG. 11 is a view showing an example of the arrangement of parallaximages;

FIGS. 12A and 12B are views showing an example of parallax imagecreation;

FIG. 13 is a view showing an example of the light-receiving areas ofmicrolenses on the light-receiving surface of the image sensor 104;

FIG. 14 is a view showing a display example on the display screen of adisplay unit 116;

FIG. 15 is a view showing an example in which all parallax images arearranged in accordance with extraction start coordinates when m=5 andn=5;

FIG. 16 is a view showing an example in which only parallax images eachhaving an average pixel value equal to or larger than a threshold arearranged and displayed on the display screen of the display unit 116;

FIG. 17 is a view showing an example of the arrangement of parallaximages;

FIG. 18 is a view showing a display example of a GUI;

FIG. 19 is a view showing an example of the arrangement of parallaximages;

FIGS. 20A to 20C are views for explaining the effect of the firstembodiment;

FIG. 21 is a schematic view of light-receiving areas on the image sensor104 when the aperture stop 102 is stopped down;

FIG. 22 is a view showing parallax images when the parallax number is5×5;

FIG. 23 is a view showing parallax images when the parallax number is3×3;

FIG. 24 is a view showing parallax images when the parallax number is7×7;

FIG. 25 is a block diagram showing an example of the functionalarrangement of a signal processing unit 106;

FIG. 26 is a flowchart of refocus evaluation information generationprocessing;

FIG. 27 is a view for explaining processing of acquiring main objectselection information;

FIG. 28 is a view for explaining central coordinates; and

FIG. 29 is a view showing a display example on a display unit 116.

DESCRIPTION OF THE EMBODIMENTS

The embodiments of the present invention will now be described withreference to the accompanying drawings. Note that the embodiments to bedescribed below are merely examples of detailed practice of the presentinvention and represent detailed examples of the arrangement in theappended claims.

First Embodiment

In this embodiment, an example will be described in which an imagesensing apparatus holding a microlens array converts a live view displayimage into an image that facilitates focusing on the display unit thatdisplays the live view display image.

Example of Arrangement of Image Sensing Apparatus

An example of the arrangement of an image sensing apparatus according tothis embodiment will be described with reference to the block diagram ofFIG. 1. Note that FIG. 1 shows an example of major components for eachprocessing to be described below, and the arrangement of the imagesensing apparatus applicable to this embodiment is not limited to thatshown in FIG. 1. That is, a component may be added to the arrangementshown in FIG. 1. Some of the units shown in FIG. 1 may be integrated.One constituent element may be decomposed into two or more constituentelements. Processing to be described later as processing to be executedby a given constituent element may be allotted to another constituentelement.

An imaging lens 101 is the main lens configured to sense an object andincludes, for example, a general zoom lens, focus lens, and blurcorrection lens used in a video camera, a still camera, or the like.

An aperture stop 102 is an optical aperture stop for the imaging lens101. The aperture stop 102 has, for example, one circular openingportion at the center, as shown in FIG. 3. The black frame portionindicates a non-opening portion formed by stopping down the aperturestop to some degree. The diameter of the opening portion (the effectivediameter of the imaging lens 101) will be referred to as D. The aperturestop 102 is arranged apart from a microlens array 103 by a distance L.

The microlens array 103 is, for example, a two-dimensional array formedby two-dimensionally arraying a plurality of microlenses in a matrix,and is arranged on the imaging plane of the imaging lens 101. Eachmicrolens has, for example, a circular or hexagonal planar shape and isformed from a solid lens, a liquid crystal lens, a liquid lens, adiffraction lens, or the like.

The imaging lens 101, the aperture stop 102, the microlens array 103,and an image sensor (image sensing element) 104 will be referred to asan image sensing unit 100 together.

FIG. 4 shows the positional relationship between the imaging lens 101,the aperture stop 102, the microlens array 103, and the image sensor104. In this case, m is the number of pixels assigned to one direction(vertical direction in FIG. 4) of each microlens of the microlens array103, s is the size (pixel size) in one side direction (verticaldirection in FIG. 4) of an image sensing element (pixel) included in theimage sensor 104, and L is the distance between the aperture stop 102and the microlens array 103, as described above. Note that the distancebetween the imaging lens 101 and the aperture stop 102 is so short as tobe negligible. For this reason, L represents the distance between theimaging lens 101 and the microlens array 103 in FIG. 4. D represents theeffective diameter of the imaging lens 101, as described above, and frepresents the distance between the microlens array 103 and the imagesensor 104.

In other words, the number m of pixels is the number of pixels in onedirection (vertical direction in FIG. 4) of an area (light-receivingarea) where the image sensor 104 receives light that has become incidentthrough one microlens. The assigned number m of pixels can be calculatedby

(m×s):f=D:L

m=D×f/(L×s)  (1)

However, only a positive integer can be adopted as m, as a matter ofcourse. Hence, if a real number is obtained as m by calculatingequations (1), the first decimal place is rounded up to an integer.

Referring back to FIG. 1, the image sensor 104 receives light that hasbecome incident through the microlens array 103 (each microlens) andacquires the light amount of the object. The image sensor 104 isarranged on the focal plane of the microlens array 103 (each microlens).The image sensor 104 includes a plurality of image sensing elementstwo-dimensionally arrayed in a matrix. A CCD (Charge Coupled Device) ora CMOS (Complementary Metal-Oxide Semiconductor) can be employed as theimage sensing element. As a matter of course, if the correspondencebetween a position on the light-receiving surface of the image sensor104 and a position on an image output from the image sensor 104 can beensured, the arrangement of the image sensor 104 is not particularlylimited.

In this embodiment, M×N pixels (image sensing elements) aretwo-dimensionally arranged in a matrix on the light-receiving surface ofthe image sensor 104. Light that has passed through one microlens isreceived in an area (light-receiving area) formed from a plurality ofpixels. The number of pixels on the light-receiving surface is, forexample, M×N=5180×3450=17871000.

Let m be the number of pixels (the assigned number of pixels for thehorizontal direction) calculated using equations (1) for the horizontaldirection of the light-receiving area, and n be the number of pixels(the assigned number of pixels for the vertical direction) calculatedusing equations (1) for the vertical direction. In this case, thenumbers m and n of pixels are related to the resolving power at anarbitrary viewpoint of a parallax image to be finally generated. Forthis reason, the resolving power at an arbitrary viewpoint of a parallaximage rises as the values m and n increase. On the other hand, (M/m) and(N/n) are related to the number of pixels (resolution) of a parallaximage. For this reason, the number of pixels of a parallax imageincreases as the values (M/m) and (N/n) increase. Hence, the number ofpixels and the resolving power at an arbitrary viewpoint of a parallaximage have a tradeoff relationship.

In addition, for example, color filters (not shown in FIG. 1) aretwo-dimensionally arranged on the pixel basis on the light-receivingsurface of the image sensor 104. The color filters have a Bayerarrangement in which filters of three primary colors, that is, red (R),green (G), and blue (B) are arranged in a checkered pattern at a ratioof R:G:B=1:2:1. Since such color filters are provided on thelight-receiving surface of the image sensor 104, pixels of a pluralityof colors corresponding to the colors of the color filters can beobtained.

An A/D conversion unit 105 converts an analog signal (an analog signal(image signal) representing the pixel value of a pixel output from eachimage sensing element) representing the light amount of the objectoutput from the image sensor 104 into a digital signal.

A signal processing unit 106 performs demosaicing processing, whitebalance processing, gamma processing, and the like for the digitalsignal output from the A/D conversion unit 105 to generate the data ofthe sensed image (sensed image data). The signal processing unit 106then generates the data of parallax images (parallax image data) basedon the pixel size s in one side direction of a pixel on the image sensor104, the distance L between the aperture stop 102 and the microlensarray 103, the effective diameter D of the imaging lens 101, and thedistance f between the microlens array 103 and the image sensor 104.Next, the signal processing unit 106 generates image data (display imagedata) to be displayed on a display unit 116 in accordance with aninstruction from an operation unit 113. The parallax image datageneration processing and the display image data generation processingwill be described later.

An encoder unit 107 performs processing of converting the parallax imagedata generated by the signal processing unit 106 into a file format suchas jpeg or mpeg.

A media interface unit 108 serves as an interface to connect a PC oranother medium (for example, a hard disk, a memory card, a CF card, anSSD card, or a USB memory).

A D/A conversion unit 109 converts the parallax image data generated bythe signal processing unit 106 into analog data.

A CPU 110 executes processing using computer programs and data stored ina ROM 111 or a RAM 112 to control the operations of the units includedin the image sensing apparatus.

The ROM 111 stores the activation program and initial setup data of theimage sensing apparatus, computer programs and data to be used by theCPU 110 to execute control the operations of the units of the imagesensing apparatus, and various kinds of information to be described fromthis embodiment as known information.

The RAM 112 has an area to temporarily store the computer programs anddata loaded from the ROM 111 and a work area to be used by the CPU 110and other units to execute processing. That is, the RAM 112 can providevarious areas as needed.

The operation unit 113 includes buttons and a mode dial. An operationinstruction input by a user operation on the operation unit 113 is sentto the CPU 110. As a matter of course, the CPU 110 may execute some ofthe functions of the operation unit 113, for example, the function ofthe mode selection button by displaying a button image on the displayscreen of the display unit 116 to be described later and causing theuser to point out the button image by a finger or the like.

An image sensing system control unit 114 controls the image sensing unit100 to do focusing, shutter opening, stop adjustment, and the like.

A character generation unit 115 generates characters, graphics, and thelike, and can generate, for example, a GUI (Graphical User Interface).The generated characters, graphics, and GUI are displayed on the displayscreen of the display unit 116 to be described later.

In general, an LCD is widely used as the display unit 116 to display thecharacters and graphics generated by the character generation unit 115,the display image generated by the signal processing unit 106, or thelike. The display unit 116 may have a touch screen function. In thatcase, some of the functions of the operation unit 113 can beimplemented, as described above. Functions other than those of theoperation unit 113 can also be implemented, as a matter of course.

An example of the arrangement of the signal processing unit 106 will bedescribed next with reference to the block diagram of FIG. 2.

A parallax number calculation unit 201 calculates the number of pixelscapable of receiving, on the image sensor, light that has passed througheach microlens. The number of pixels capable of receiving, on the imagesensor, light that has passed through each microlens will be referred toas a parallax number hereinafter.

An LF creation unit 202 creates the light field of the image sensingapparatus based on the parallax number. The light field represents thecorrespondence between the passage position of a light beam through theimaging lens and the receiving position of the light on an image sensingelement.

An image reconstruction unit 203 rearranges the pixels of the sensedimage data held in a buffer 204 based on the light field, therebygenerating parallax image data observed from a different viewpoint.

A parallax image extraction unit 205 selects and extracts the parallaximage data based on the display mode information of the operation unit113 and transmits the parallax image data to the encoder unit 107 andthe D/A conversion unit 109.

<Operation of Image Sensing Apparatus>

The operation of the image sensing apparatus according to thisembodiment will be described next. Light (object image) that has becomeincident on the microlens array 103 through the imaging lens 101 and theaperture stop 102 forms an image on the light-receiving surface of theimage sensor 104 in accordance with the shape of each microlens. Thatis, an area (light-receiving area) where the light that has passedthrough a microlens is received is formed for each microlens on thelight-receiving surface of the image sensor 104. FIG. 5 shows an exampleof light-receiving areas for the respective microlenses on thelight-receiving surface of the image sensor 104.

Since the opening portion of the aperture stop 102 is circular, as shownin FIG. 3, a circular light-receiving area is formed on thelight-receiving surface of the image sensor 104 in correspondence witheach microlens. At this time, light beams incident on the microlensarray 103 are received at different positions on the light-receivingsurface of the image sensor 104 in accordance with their incidencepositions on the imaging lens 101.

Each image sensing element of the image sensor 104 outputs an analogsignal corresponding to the received light amount. As a result, theimage sensor 104 outputs an analog signal for each pixel. The A/Dconversion unit 105 converts (A/D-converts) the analog signal of eachpixel into a digital signal, thereby generating a digital signal foreach pixel. This A/D conversion can be general processing. For example,the light amount of the object is photoelectrically converted into asignal and then converted into a digital signal representing a 14-bitdigital value.

Next, the signal processing unit 106 performs demosaicing processing fora general Bayer arrangement, white balance processing, and gammaprocessing for the digital signal of each pixel converted by the A/Dconversion unit 105 to generate sensed image data. The signal processingunit 106 reconstructs the sensed image data to parallax image data basedon the light field, and generates display image data to be displayed onthe display unit 116. The parallax image data generation processing andthe display image data generation processing will be described withreference to the flowchart of FIG. 6.

In step S601, the parallax number calculation unit 201 acquires imagesensing unit parameters from the image sensing unit 100. The imagesensing unit parameters are the pixel size s in one side direction of apixel on an image sensing element, the distance L between the aperturestop 102 and the microlens array 103, the effective diameter D of theimaging lens 101, and the distance f between the microlens array 103 andthe image sensor 104. Acquiring the image sensing unit parameters fromthe image sensing unit 100 has been described above. However, forexample, the image sensing unit parameters may be stored in the ROM 111in advance, and the parallax number calculation unit 201 may acquire theimage sensing unit parameters from the ROM 111. Alternatively, aparameter input field may be displayed on the display unit 116, and thevalues the user inputs to the parameter input field by operating theoperation unit 113 may be used as the image sensing unit parameters.Otherwise, for example, an effective diameter D0 of the aperture stop102 in the full-aperture state may be stored in the ROM 111. When anF-number Fnum of the stop is obtained, the effective diameter D may beobtained by calculating

D=D0/Fnum  (2)

The distance L between the aperture stop 102 and the microlens array 103or the distance f between the microlens array 103 and the image sensor104 may be detected using a radar device provided for distancemeasurement. Note that the image sensing unit parameters may bereacquired when, for example, the stop or the distance between theimaging lens 101 and the microlens array 103 or the image sensor 104 haschanged.

In step S602, the parallax number calculation unit 201 calculates theparallax number based on the image sensing unit parameters acquired instep S601. FIG. 4 shows the image sensing unit parameters and thepositional relationship between the imaging lens 101, the aperture stop102, the microlens array 103, and the image sensor 104. At this time,the number m of elements for receiving light from each microlens isgiven by

$\begin{matrix}{{{m \times s\text{:}f} = {D\text{:}L}}{m = \frac{Ls}{Df}}} & (3)\end{matrix}$

where m is a positive integer.

The number m×n of elements for receiving light on the image sensingelements is calculated from equations (3). The parallax numbercalculation unit 201 sends the number m×n of elements to the LF creationunit 202 as the parallax number of the image sensing apparatus accordingto this embodiment. Note that the parallax number calculation is merelyan example of one form of the image sensing apparatus holding a lensarray, and the present invention is not limited to this. For example,the parallax number may be acquired from the ROM 111. When the imagesensing unit parameters are reacquired, the parallax number is alsorecalculated.

In step S603, the LF creation unit 202 divides the imaging lens 101 intoareas based on the parallax number and assigns an index to each area.The number of elements for receiving light for each microlens, which iscalculated by the parallax number calculation unit 201, is the parallaxnumber. Hence, when the parallax number is m×n, the imaging lens isdivided into m×n areas, and indices ML(1,1), ML(2,1), . . . , ML(m,n)are assigned to the areas, respectively. For example, FIG. 7 illustratesan example of area division and the indices of areas of the imaging lens101 when m=3, and n=3.

In step S604, based on the image sensing unit parameters and areadivision of the imaging lens, the LF creation unit 202 creates a lightfield that is a list in which the correspondence between coordinates(x,y) of each image sensing element and coordinates (u,v), on theimaging lens 101, of a light beam that becomes incident on the imagesensing element is registered. As shown in FIG. 8, the coordinates (u,v)on the imaging lens are defined on a coordinate system including theimaging lens while placing the origin at the center of the imaging lens.For example, assume that the u-v coordinate system ranges from −1 to 1,as shown in FIG. 8. The created light field is a table of thecoordinates (u,v) corresponding to the coordinates (x,y) of each pixeland the index of the divided area including the coordinates (u,v), asshown in FIG. 9. For the light field of the image sensing apparatus, aline is drawn from a pixel to the center of a microlens, as shown inFIG. 10, thereby acquiring the coordinates (u,v) of the passage point onthe imaging lens as the coordinates corresponding to the pixel. Thisoperation is performed for all pixels to create the light field. Notethat the light field shown in FIG. 10 is merely an example for thepositional relationship of the components in the image sensing unit 100.The present invention is not limited to this as far as the light fieldrepresents the correspondence between the coordinates (u,v) on theimaging lens 101 and the coordinates (x,y) of the pixel on the imagesensor 104. The light field may be recreated when the distance L betweenthe aperture stop 102 and the microlens array 103 or the distance fbetween the microlens array 103 and the image sensor 104 in the imagesensing unit 100 has changed.

In step S605, the digital signal input from the A/D conversion unit 105is stored in the buffer 204 as sensed image data.

In step S606, the image reconstruction unit 203 reconstructs the sensedimage data held in the buffer 204 into parallax image data based on thelight field obtained from the LF creation unit 202. More specifically,the pixels in the sensed image data are rearranged so that u of thecoordinates (u,v) in the light field increases from left to right, and vincreases from above to below. That is, when the coordinates of a pixelafter rearrangement are represented by (x′,y′), rearrangement is done by

(x′,y′)=(u,v)  (4)

This allows to reconstruct parallax image data having parallaxes as manyas the divided areas of the imaging lens. Parallax images indicate animage group with parallaxes including, when an object as shown in FIG.12A is sensed, the image of the object sensed from the upper side withrespect to the center and the image of the object sensed from the rightside, as shown in FIG. 12B.

In step S607, the image reconstruction unit 203 stores the reconstructedparallax image data in the buffer 204. If parallax image data is alreadystored in the buffer 204, the image reconstruction unit 203 updates thealready stored parallax image data. The image reconstruction unit 203also stores the light field in the buffer 204.

In step S608, the image reconstruction unit 203 determines whether thereis a change in the image sensing unit parameters of the light field.Upon determining that there is no change, the updating is determined tobe completed, and the process advances to step S609. Upon determiningthat there is a change in the image sensing unit parameters, the processreturns to step S601.

In step S609, the image reconstruction unit 203 performs overlapdetermination in the light field. In the overlap determination, it isdetermined whether the imaging lens has two or more kinds of coordinatescorresponding to each pixel. Upon determining that overlap exists, theprocess advances to step S610. Upon determining that no overlap exists,the process advances to step S611. For example, assume that the imagesensor 104 has light-receiving areas in the state shown in FIG. 13. Asshown in FIG. 13, when each light-receiving area is wide, overlap areasare generated.

In step S610, the image reconstruction unit 203 gives overlapinformation representing existence of overlap to a light fieldcorresponding to a pixel determined as overlap and updates the lightfield in the buffer 204. In step S611, the parallax image extractionunit 205 confirms the currently set display mode. The user can set anyone of the display modes by operating the operation unit 113. Morespecifically, when the user inputs an instruction to select a displaymode using the operation unit 113, the character generation unit 115generates a display mode selection screen (GUI) and displays it on thedisplay screen of the display unit 116. FIG. 14 shows a display exampleon the display screen of the display unit 116.

The 3×3 matrix shown as a GUI corresponds to 3×3 parallax images. Forexample, pointing out the rectangle at the upper left corner using theoperation unit 113 makes it possible to designate a parallax imageML(−1,−1) obtained by sensing the object from upper left. In this way,pointing out a rectangle ith (1≦i≦3 in FIG. 14) rightward from upperleft and jth (1≦j≦3 in FIG. 14) downward using the operation unit 113allows to designate a parallax image ML(−2+i,−2+j). The parallax imagedesignation method and the arrangement of the GUI therefor are notlimited to those described above, as a matter of course.

When the user points out one rectangle using the operation unit 113 aparallax image corresponding to the pointed position is designated, anda 1-parallax display mode is set. When the user instructs to select allrectangles (all rectangles of the GUI) using the operation unit 113, anall-parallax display mode is set. Note that the method of setting the1-parallax display mode or the all-parallax display mode is not limitedto this. Alternatively, for example, one of the all-parallax displaymode and the 1-parallax display mode may be selected using a check boxgenerated by the character generation unit 115. In the above-describedexample, two display modes are selectable. However, the presentinvention is not limited to this. There may also exist a mode to selectand display several parallax images.

Anyway, when the user sets a display mode using the operation unit 113,data representing the set display mode (when the 1-parallax display modeis set, the data includes data representing the designated parallaximage) is written in the RAM 112. Hence, in step S611, the data writtenin the RAM 112 to represent the display mode is referred to, and whichdisplay mode is represented by the data is determined.

Upon determining in step S611 that the 1-parallax display mode is set,the process advances to step S612. When the all-parallax display mode isset, the parallax image extraction unit 205 directly arranges theparallax image data for each parallax, as shown in FIG. 11, and outputsit to the encoder unit 107 and the D/A conversion unit 109 as displayimage data.

How to arrange and display the parallax images is not limited to thatdescribed above, as a matter of course. For example, when the parallaximages are arranged as shown in FIG. 11, some of the parallax images maybe extracted and arranged as display images. A parallax image having anaverage pixel value equal to or smaller than a given threshold may beexcluded from the display target because it is “too dark”.

FIG. 15 illustrates an example in which all parallax images are arrangedin accordance with extraction start coordinates when m=5 and n=5. Atthis time, when the opening portion of the aperture stop 102 iscircular, as shown in FIG. 3, non-light-receiving areas where lightreceiving does not occur are generated on the image sensor 104, as shownin FIG. 5. Referring to FIG. 15, the parallax images (hatched portionsin FIG. 15) to be listed below are formed from pixels extracted from thenon-light-receiving areas and are therefore darker than the remainingparallax images.

Display image data areas ML(1,1), ML(2,1), ML(4,1), ML(5,1), ML(1,2),ML(5,2), ML(1,4), ML(5,4), ML(1,5), ML(2,5), ML(4,5), and ML(5,5)

Only parallax images each having an average pixel value equal to orlarger than a threshold may be arranged, as shown in FIG. 16, anddisplayed on the display screen of the display unit 116. A dark parallaximage may be displayed after made lighter by multiplying each pixelvalue by a gain. Alternatively, parallax images in the neighborhood maybe added.

Alternatively, as shown in FIG. 17, the parallax images may beextracted, arranged and enlarged to 3×3, and thus displayed on thedisplay screen of the display unit 116. A parallax image given overlapinformation in step S610 may be excluded from the display target. For aparallax image given overlap information, “information representingoccurrence of overlap” may be displayed in place of the parallax image,as shown in FIG. 19.

Referring back to FIG. 6, in step S612, the parallax image extractionunit 205 determines whether to refer to the “data representing thedesignated parallax image” included in the data representing the1-parallax display mode. Whether to refer the data can be either presetor selected by the user. To refer to the data, the process advances tostep S613. Not to refer, the process advances to step S614. When aninstruction to select the 1-parallax display mode is simply input usinga GUI than the above-described matrix GUI, there exists no datarepresenting a designated parallax image. In this case, the processadvances to step S614.

In step S613, the parallax image extraction unit 205 enlarges thedesignated parallax image as needed, and outputs the enlarged parallaximage data to the D/A conversion unit 109 as display image data. Forexample, in FIG. 11, when the rectangle at the upper left corner ispointed out on the 3×3 matrix GUI of the above-described example, theparallax image data ML(1,1) is extracted, enlarged as needed, and outputto the D/A conversion unit 109. A known enlarging method such as abicubic method is usable. The scaling factor of enlargement iscalculated from the number of pixels of the display screen of thedisplay unit 116. The enlargement processing is not essential, andreduction may be performed in place of enlargement, as a matter ofcourse.

The operation on the GUI (switching the display target parallax image)may be done during display image display on the display unit 116. Forexample, when the user points out another parallax image using theoperation unit 113 during display of a display image on the display unit116, a display image is generated from the designated parallax image andoutput to the D/A conversion unit 109.

Note that the display configuration on the display screen of the displayunit 116 is not limited to the above-described example. For example, aGUI as shown in FIG. 18 may be displayed on the display unit 116 todisplay a display image on the upper side of the display unit 116 and aparallax image on the lower side and cause the user to select a parallaximage on the lower side. In the GUI shown in FIG. 18, each parallaximage data generated in step S607 is reduced and arranged on the lowerside of the display unit 116, and the selected parallax image isenlarged and displayed on the upper side. Selection of a parallax imagegiven overlap information may be prohibited. The parallax image data maydirectly be arranged for each parallax, as shown in FIG. 11, and outputto the encoder unit 107 and the D/A conversion unit 109 as display imagedata.

In step S614, the parallax image extraction unit 205 enlarges an area MLlocated at the center of the parallax image data, as needed, as in stepS613, and outputs the enlarged parallax image to the D/A conversion unit109 as a display image. The enlargement processing is not essential, andreduction may be performed in place of enlargement, as a matter ofcourse.

Note that the display unit 116 displays the display image data receivedfrom the D/A conversion unit 109 as a display image on, for example, anLCD.

As described above, according to this embodiment, the user can confirmvia the display unit 116 whether a parallax image acquired by the imagesensing apparatus including the microlens array 103 is in focus. Forexample, when the object is a planar chart as shown in FIG. 20A, adisplay image as shown in FIG. 20B is conventionally obtained throughthe microlens array 103. Hence, it is impossible to confirm whether theplanar chart is in focus. However, according to this embodiment, theimage of the center viewpoint or the like is enlarged and displayed, asshown in FIG. 20C. This allows to confirm whether the planar chart is infocus.

Second Embodiment

The series of processes described in the first embodiment are performedat a predetermined time interval to update the display image in realtime, thereby implementing live view display. In addition, every time anassigned pixel number calculation parameter or the lens position isupdated, the above-described processing is performed. This makes itpossible to reflect the influence of the focal length, the aperturestop, and the movement of the lens and image sensing elements uponshooting on the display image in real time. For example, when the focallength or an aperture stop 102 changes, the light-receiving area on animage sensor 104 changes.

FIG. 21 is a schematic view of light-receiving areas on the image sensor104 when the aperture stop 102 is stopped down. FIG. 13 is a schematicview of light-receiving areas when the aperture stop 102 is opened. Atthis time, since the image sensing unit parameter changes, the procedureof a signal processing unit 106 is performed from step S601 again tocreate a new light field and thus cope with the change. For example,when the aperture stop 102 is stopped down, the light-receiving areasbecome smaller, as shown in FIG. 21. Hence, the parallax number m×ndecreases as can be seen from equations (2) and (3). Assume that theparallax number of parallax image data is 5×5, and this decreases to 3×3when the aperture stop is stopped down. At this time, the light field isrecreated. The parallax number of the parallax image data decreases fromthat in FIG. 22 to that in FIG. 23. As described above, even when thelight-receiving areas are narrowed by, for example, stopping down theaperture stop 102 or increasing a distance L, the parallax image datacan correctly be generated.

When the aperture stop 102 is opened to widen the light-receiving areason the image sensor 104, the parallax number m×n increases as isapparent from equations (2) and (3). For example, when the parallaxnumber increases from 5×5 to 7×7, parallax image data as shown in FIG.24 is obtained. As described above, even when the light-receiving areasare widened by, for example, opening the aperture stop or shortening thedistance L, the parallax image data can correctly be generated.

In the above-described embodiment, the in-focus state can be confirmedby displaying a parallax image. However, the image can also be used foranother purpose, as a matter of course. For example, contrast AFprocessing may be done from a CPU 110 and an image sensing systemcontrol unit 114 using a display image.

The processing has been described above concerning an image sensingapparatus which can change the focal length and move the aperture stop102, the imaging lens 101, and the image sensor 104. However, this doesnot apply in an image sensing apparatus that uses a single focal pointor a single aperture stop and does not change the size of eachlight-receiving area on the image sensor 104. In this image sensingapparatus, the light field may be held in a ROM 111 or the like inadvance because it is not recreated from the image sensing unitparameters. The same processing can be done even in an image sensingapparatus which does not change the light field because of thepositional relationship in an image sensing unit 100 regardless of theaperture state of the aperture stop. Processing after light fieldcreating is the same as that from step S605.

Third Embodiment

In this embodiment, an example will be explained in which display isperformed to allow the user to easily confirm whether focus adjustmentprocessing (to be referred to as refocus hereinafter) after shooting,which is to be performed by an image sensing apparatus holding a lensarray, is possible. Points different from the first embodiment willmainly be described below.

Example of Arrangement of Image Sensing Apparatus

FIG. 25 is a block diagram showing an example of the functionalarrangement of a signal processing unit 106 according to thisembodiment. The signal processing unit 106 includes a parallax numbercalculation unit 201, an LF creation unit 202, an image reconstructionunit 203, a buffer 204, a parallax image extraction unit 205, a mainobject extraction unit 2501, and a parallax amount calculation unit2502.

The parallax number calculation unit 201, the LF creation unit 202, theimage reconstruction unit 203, and the buffer 204 are the same as in thefirst embodiment, and a description thereof will not be repeated.

The parallax image extraction unit 205 extracts a parallax image fromthe center viewpoint of parallax image data stored in the buffer 204,enlarges the parallax image, and sends it to a D/A conversion unit 109.

The main object extraction unit 2501 acquires the information of themain object designated by the user via an operation unit 113 or thetouch panel of a display unit 116, and extracts the main object fromeach parallax image data.

The parallax amount calculation unit 2502 calculates the barycentriccoordinates of the main object on each parallax image coordinate systemfrom the main object extracted from each parallax image data, andcalculates the parallax amount based on the distance between thebarycentric coordinates and those of the parallax image of the centerviewpoint. Next, it is determined based on the parallax amount whetherthe main object has a parallax sufficient for refocus. Then, refocusdetermination information is generated from each parallax imageincluding the main object and its parallax amount, gives the informationto the parallax image, and transmits it to the D/A conversion unit 109.

<Operation of Image Sensing Apparatus>

The operation of the image sensing apparatus according to thisembodiment will be described next.

Processing up to sensed image data generation of the signal processingunit 106 is the same as in the first embodiment, and a descriptionthereof will not be repeated. Refocus evaluation information generationprocessing will be described below with reference to the flowchart ofFIG. 26.

The processes of steps S2601 to S2610 are the same as those of stepsS601 to S610 described above, and a description thereof will not berepeated.

In step S2611, the parallax image extraction unit 205 extracts theparallax image of the center viewpoint of the parallax image data,enlarges the parallax image, and transmits it to the D/A conversion unit109. In addition, a message to cause the user to select the main objectis displayed on the display unit 116.

In step S2612, the main object extraction unit 2501 acquires main objectselection information from the operation unit 113. The main objectselection information is a point (x″c,y″c) in the x″−y″ coordinatesystem of the parallax image data of the center viewpoint, which isselected by the user using the operation unit 113 from the parallaximage of the center viewpoint displayed on the display unit 116 in stepS2611. For example, assume that a face image is sensed in the parallaximage data of the center viewpoint, as shown in FIG. 27. When the userdesignates the point indicated by a full circle and designates the faceas the main object, the x″y″ coordinates of the full circle are storedin a RAM 112 as the main object selection information. The main objectextraction unit 2501 acquires this information.

In step S2613, the main object extraction unit 2501 acquires theparallax image data of the center viewpoint from the buffer 204,extracts the object based on the main object selection information, andcalculates the center coordinates (x″0,y″0) of the extracted object. Thecenter coordinates are obtained by surrounding the object by a rectangleand defining the center of the rectangle, as shown in FIG. 28. Note thatobject extraction can be done by a known method.

In step S2614, the main object extraction unit 2501 acquires parallaximage data other that of the center viewpoint from the buffer 204, andextracts the object extracted in step S2613 from each parallax imagedata. Note that extraction can be done by a known method such as patternmatching. Next, the center coordinates (x″1,y″1), (x″2,y″2), (x″3,y″3),. . . of the object are calculated for the respective parallax images,as in step S2613. The main object extraction unit 2501 then sends thecenter coordinates of all parallax images from which the object can beextracted to the parallax amount calculation unit 2502.

In step S2615, the parallax amount calculation unit 2502 calculates theparallax amount of each parallax image including the object designatedby the user. For example, a parallax amount E1 of the parallax imagehaving the object center coordinates (x″1,y″1) is calculated by

E1=√{square root over ((x″1−x″0)²+(y″1−y″0)²)}{square root over((x″1−x″0)²+(y″1−y″0)²)}  (5)

In a similar manner, parallax amounts E2, E3, . . . are calculated forthe parallax images from which the object can be extracted.

In step S2616, the parallax amount calculation unit 2502 generatesrefocus evaluation information based on each parallax amount calculatedin step S2615, and gives the refocus evaluation information tocorresponding parallax image data. All parallax image data aretransmitted to the D/A conversion unit 109 as display image data. Therefocus evaluation information is 1-bit information O or x. When theobject exists in the parallax image, and the parallax amount of theparallax image data is larger than a threshold Th necessary andsufficient for refocus, refocus evaluation information O is given to theparallax image data. When the parallax amount is smaller than thethreshold Th, refocus evaluation information x is given to the parallaximage data. The refocus evaluation information determination conditionis not limited to that described above. The threshold Th may beincreased as the parallax image data is farther apart from the parallaximage data of the center viewpoint. In the above-described example, onlythe scalar quantity of the parallax amount is used as the determinationcondition. However, a direction may be combined. For example, whenparallax image data is located above the parallax image data of thecenter viewpoint, the parallax amount vector of the object needs to beupward. Hence, whether the vector is upward may be determined.

Next, the display unit 116 displays display image data on, for example,an LCD as a display image. The images may be displayed in, for example,the all-parallax mode, as shown in FIG. 29. At this time, the refocusevaluation information O or x is displayed at the upper right of eachparallax image based on the refocus evaluation information given in stepS2616. The display method is not limited to this. Only parallax imagesgiven the refocus evaluation information may be displayed on the displayunit 116 one by one, and switched as the time elapses. At this time, therefocus evaluation information of the parallax image is displayedsimultaneously.

A display image is displayed on the display unit 116 by performing theabove-described processing. This allows the user to easily confirm viathe display unit 116 whether the main object of the sensed imageacquired by the image sensing apparatus including a microlens array 103has a parallax amount sufficient for refocus to be performed later.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (for example, computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application Nos.2010-282397 filed Dec. 17, 2010 and 2011-251024 filed Nov. 16, 2011,which are hereby incorporated by reference herein in their entirety.

1.-6. (canceled)
 7. An image sensing apparatus comprising: an imageacquisition unit that acquires a sensed image generated from an imagesignal output from an image sensor for receiving light that becomesincident sequentially through an imaging lens and a microlens array thatis a two-dimensional array including a plurality of microlenses, whereineach of the plurality of microlenses corresponds to any one of pixelgroups in the image sensor and each pixel in the pixel group receivesthe light that has passed through corresponding microlens; an parameteracquisition unit that acquires an image sensing unit parametercorresponding to the sensed image; and a generation unit that generatesa display image including a plurality of parallax images based on theimage sensing unit parameter, wherein the parallax images are generatedfrom the sensed image and the number of parallax images included in thedisplay image is determined based on the number of pixels included inareas, corresponding to the plurality of microlenses, in the sensedimage.
 8. The image sensing apparatus according to claim 7, furthercomprising a calculation unit that calculates the number of pixelsincluded in areas, corresponding to the plurality of microlenses, in thesensed image by using the image sensing unit parameters, wherein thedisplay image is generated based on the calculated number of pixels. 9.The image sensing apparatus according to claim 7, further comprising acontrol unit that controls a configuration of an optical systemincluding the image sensor, wherein the parameter acquisition unitacquires the image sensing unit parameter that corresponds to the sensedimage acquired on the basis of the configuration changed under controlof the control unit, and the generation unit generates the display imagebased on the image sensing unit parameter acquired by the parameteracquisition unit.
 10. The image sensing apparatus according to claim 7,wherein the number of parallax images included in the display image isequal to the calculated number of pixels.
 11. The image sensingapparatus according to claim 1, wherein, when the sensed image has apixel that corresponds to a plurality of microlenses, the generationunit generates the display image so that the display image does notinclude a parallax image that includes a pixel corresponding to aplurality of microlenses.
 12. The image sensing apparatus according toclaim 1, wherein the generation unit generates the display image so thatthe display image does not include a parallax image whose an averagepixel value is below a threshold.
 13. A method for controlling an imagesensing apparatus including an image sensor for receiving light thatbecomes incident sequentially through an imaging lens and a microlensarray that is a two-dimensional array including a plurality ofmicrolenses, wherein each of the plurality of microlenses corresponds toany one of pixel groups in the image sensor and each pixel in the pixelgroup receives the light that has passed through correspondingmicrolens, the method comprising: a step of acquiring an image sensingunit parameter corresponding to a sensed image generated from an imagesignal output from the image sensor; and a step of generating a displayimage including a plurality of parallax images based on the imagesensing unit parameter, wherein the parallax images are generated fromthe sensed image and the number of parallax images included in thedisplay image is determined based on the number of pixels included inareas, corresponding to the plurality of microlenses, in the sensedimage.